On the flexibility of complex systems 
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Many complex systems satisfy a set of constraints on tlieir degrees of freedom, and at the same 
time, tliey are able to work and adapt to different conditions. Here, we describe the emergence 
of this ability in a simplified model in which the system must satisfy a set of random dense linear 
constraints. By statistical mechanics techniques, we describe the transition between a non-flexible 
system in which the constraints are not fully satisfied, to a flexible system, in which the constraints 
can be satisfled in many ways. This phase transition is described in terms of the appearance of zeros 
modes in the statistical mechanics problem. 
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Much of the complexity of biological and social systems 
is rooted in their ability to function in a large variety of 
conditions. At the same time, complex systems must sat- 
isfy real constraints. For example, the metabolic network 
to function must, at least, satisfy the steady state equa- 
tions of the intermediate metabolites production 0, , 
and a financial market to be efficient must have a zero ex- 
cess demand for any distinguishable state of the market 
0. However, to preserve this flexibility, the constraints 
can not completely determine the state of the system. 
This will be fully defined by the external conditions, some 
optimization principle, and/or the previous history of the 
system. 

In the case of metabolic networks, all steady states of 
fluxes {s} are defined by the system A • s — g = such 
that < Si < s™°^ are intervals that define a convex 
manifold r2. The matrix A in this problem is the M x N 
stoichiometric matrix of all the reaction in the network, 
with M metabolites and N metabolic fluxes, while the 
M dimensional vector g is different from zero only for 
metabolites in input or in output to the network. The 
real state of the cell is required to be one of these steady 
states 0,0. In addition, it is usually assumed that the 
actual state optimizes the cell growth rate, but this as- 
sumption have to be relaxed in various cases Q . 

This description of the problem is reminiscent of the 
Theory of Linear Programming Q , where one must op- 
timize a linear function of unknown variables subject to 
linear constraints. Like in metabolic networks, the con- 
straints define the space of possible solutions, while the 
optimization function, chooses a given one. 

A similar problem appears in the study of the Minor- 
ity Game (MG) 0, 0] . The Minority Game is a simple 
model that was proposed recently to mimic the dynamics 
of agents in a market. In one of the version of this game, 
the Gran Canonical Minority Game 7], the traders in 
the market, at each time step have access to an external 
information /i — l,...,Af which describes the market 
state. Each agent i is provided with a single strategy 
indicating which action (buy /sell) to take given the in- 
formation fi, i.e. = ±1 . There are two types of 



agents in the markets: Ng speculators that can decide to 
trade or not to trade depending on their expected suc- 
cess and Np producers which always play their strategy. 
This game has been approached by statistical mechanics 
0, techniques and it has been proved that the dy- 
namics described above minimizes the Lyapunov func- 
tion = af - B^'^ /M. The soft variable 

Si e (0, 1) corresponds to the average use of the strat- 
egy of agent i in the stationarity state of the game and 



indicates the information injected by 
the producers in the market. Below a critical value of 
a = M/N the systems is efficient, i.e. it is such that the 

expected excess demand Eiii '^i^i ~ = for each 
information /i = 1, . . . , M and H — 0. 

It becomes then transparent that a very important 
question in the study of complex systems is to know how 
fiexible the system is, i.e. how large is the volume V 
of the possible solutions as a function of the parameters 
of the equations. In the examples described above, it 
is equivalent to find the number of steady states of the 
metabolic fiuxes in a cell or the number of agent actions 
which clear a financial market. 

Technically the problem can be cast in terms of finding 
the number of solutions of a linear system of equations 
when the variables are constrained in a connected finite 
volume f2. Unfortunately, it is well known that the enu- 
meration of all the vertex of the convex set containing 
these solutions is a complete problem . In a more 
general context, the calculation of the volume of a com- 
plex polyhedron in the N dimensional space, in the so 
call representation (see for example ^^), is expressed 
by a set of M linear inequalities A • s > 0, where A is 
again a M x N matrix. 

Similar constraint satisfaction problems have been suc- 
cessfully treated by statistical mechanics techniques. In 
the case of boolean variables it is known that there is 
a transition between satisfiable to unsatisfiable (SAT- 
UNSAT) instances as a function of the ratio a between 
the number of equations M and variables N. Through 
statistical mechanics methods it has been proven 
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0, 01 that in many A^P-complete problems this SAT- 
UNSAT transition is preceded by a dynamical transition 
identified with the absence of a replica symmetric solu- 
tion and with the clusterization of the phase space. On 
the contrary if the problem is defined on continuous vari- 
ables, no general approach describing the transition is 
known. Usually, the solution of the volume of the satis- 
fied space is found by imposing a spherical constraint on 
the variables [3 ■ However, as it has been recently shown 
in the Minority Game context [Tsl l , in many situations the 
spherical constrain generates a different scenario, and in 
addition, it is often not justified. 

In this paper, using statistical mechanics methods, we 
describe, as a function of a = M/N, the transition be- 
tween satisfiable and unsatisfiablc dense linear systems 
of equations defined on continuous variables. In fact it is 
intuitively clear that there is a specific value of ac such 
that for a > ac the system is not able to satisfy all the 
constraints while for a < ac the system becomes satisfi- 
able for different values of the variables. Here we deter- 
mine the volume of the space of these solutions and the 
average values of the variables in term of their admitted 
values, and the statistical properties of the equations. 

For simplicity, we will consider N variables and M 
equations with random coefficients chosen from a dis- 
tribution P(^) with zero average (^) = and with vari- 
ance (^^) = The system which we would like to be 
satisfied is 



= l,aM. 



(1) 



where represents the inhomogeneities of the equations, 
i.e. for example the input or output fluxes in metabolic 
networks. The continuous variables variables {si} would 
be defined over a set fl = uj^ with Si G w and \uj\ — L. 
In the case of convex space we would like to address 
which are the typical features of a generic problem with 
N variables and M equations for a generic matrix ((i^f )). 

Using a Gaussian representation for the delta functions 
Ylfj_S{J2i^iSi — g^) imposed by the hard constrains ^ 
we get the following definition of the unormalized volume 
of solutions 

to be taken in the limit /3 oo. Conse- 

quently we have introduced a Gaussian measure /o = 

n^e-'^(S-«^--»'')V((7r//3)*'^/^V'(/3) in the space of the 
variables fi. 

This kind of problem, within the formalism of statis- 
tical physics, can be approached defining a Hamiltonian 
H to be minimized when the system is solved. For our 
case of interest H is given by 



which is equivalent to the one defined in the study of the 
Minority Game 

For such a system the partition function reads 



Z = 



W_dsi 



exp 



(4) 



(3) 



and the entropy is given by 



S=-< log(/o) >=I3E+— log (-] + log(F) (5) 



Thus our goal would be to evaluate the free energy 
of this system mediated over the quenched disorder 
^(log(Z)) ^ F = E - TS, being E the energy and 
S the entropy of the system. Obviously, from our Hamil- 
tonian we must have, E ~ for satisfiable systems 
and E for unsatisfiablc systems. 

The calculation of (log(Z)) proceeds using the usual 
replica trick (log(Z)) — lim„^o ^^—^ — - in which Z" indi- 
cates the n times replicated partition function with the 
same quenched noise. Then, using a Replica Symmet- 
ric ansatz 16], and on similar lines to the analogous 
Hamiltonian of the Minority Game 0, @ we obtain a 
result, which, extended to the limiting case L — > oo, 
(3 ^ oo (which is the Algebra limit) provides an en- 
ergy E/N = - 1) + ^ and an entropy S/N = 

-f log a + log (a - 1) -f i log . This solution 

is wrong for a < 1. In fact the energy becomes negative, 
contradicting equation Moreover for a < 1 the en- 
tropy must acquire a term of the order of log(L) that 
here is not present. This signals that the standard RS 
ansatz in the L, /? — s- oo fails for a < 1. 

In order to solve this problem, which arises also in the 
finite L case, we gain insight from the known algebra 
solution of the case L ^ oo. In this case the linear sys- 
tem of equations is solved first fixing, arbitrarily, A'^ — M 
variables and then solving respect to the others. Con- 
sequently, to solve the problem at finite L we assume 
that although all the variables are equivalent, below the 
satisfiable transition an arbitrary set of (1 — m)N vari- 
ables is free to assume any value Si € uj without fiuc- 
tuations while the other mN variables can still satisfy 
the linear problem. Then, a new variational parame- 
ter m is introduced in the free energy and fixed through 
the saddle point equations. The constrained variables 
i — l,...mN would have an overlap Qa,b =< s'^s^ > 
and conjugated variables Qa,b Replica Symmetric, i.e. 

Qa,b = QSa,b + - Sa,b), Qa,b = Q5a,b + ^(l " 5a,b) 

On the contrary the other (1 — m)N variables would 
have and average X]i=mAr -^i — {^ — m)NS°', and an over- 
lap T,ZmN sf4 = (1 - m)NS''S\ Where S'' ^ S and 
S"^ = S, keeping the RS scenario. 

Using this ansatz the free energy reads, 
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(1 - 771)5* S* - m{QQ - qq) + — 



-^e-^'/2 In i?L(g, Q) - (1 - m) InL - (1 - m) In (l{S)) 
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(6) 



and the saddle point equations become 



mq 



{q-q) 



Q-q 



2 {Q~i+2d^y 



2Q-q' 1 



2/3 cr^m 



1 



(7) 



{{y^)) 



DylnHL{Q,q) = \n (lI{S)) + SS - [Q - q)S^ 



to be considered together with the two equations 
df 



dS 
55* 



i.e. S = 2S{Q~q) 
i.e. S = {s)g 



(8) 



with Hl{QA) = 

I dsse-^V dse-^'^ and finally ^ ^^(.9^)7^. 

If the interval lo is symmetric respect to zero, the so- 
lution S* = 5* = is always allowed. When w is not 
symmetric respect to zero it is always possible to trans- 
late the variables and to define the problem on a sym- 
metric interval around zero. In this translation the vari- 
ables change following Si — > Si — xq and the constrain 
following 5^ + o.'^xq (g^ ^ 3^ + f^a^o) '^i*^ 

xo indicating the center of mass of the interval lj. In 
the special case of the algebra limit, L,f3 oo, this 
solution for a < 1, predicts: the free energy /?/ = 

-f log - (1 - a) InL, the energy E/N = ^ and 

the entropy S/N = f log + {1 - a) InL. Conse- 

quently for a < 1 the system is satisfiable in the limit 
/3 — !■ oo, in fact E ^ and the volume of solutions 

V given by e<'°s(^)> = L^-'^^ {-^Y''^ as it should since 

M/2 

is the determinant of a Af x M matrix of co- 
efficients with {Ci = and variance {(^) = a^/N 

In the case in which w is bounded and L finite, from 
© and jnjl, we have for the volume of solutions, 
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FIG. 1: The volume of solutions of a linear system of equa- 
tions Q — 1/3. The continuous line corresponds to the the- 
oretical prediction, and the symbols to the simulations with 
iV = 6 ( □), = 9 (<). 
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(InV) = liiL + m{QQ-qq) 
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2 Q~q+ 



fin 



2TTma [Q-q 



2/3 cr^m 



(9) 



with the saddle point equations defining the value of the 
variational parameters. In this case, the saddle point 
equations ((TJ must be solved numerically and the validity 
of our ansatz may be tested with simulations. 

On the other hand, to estimate the average values of 
the variables one must consider that while the equations 
(0), (jHJ allow for a solution with tti < 1 the system is 
satisfiable and a series of 1 — to zeros modes are present 
in the solution. These zero modes are associated to the 
freedom in which the (1 — m)N variables can be chosen. 
In this case, as a fraction (1 — m) of free variables can 
assume any value in the set uj the average value of the 
variables (s) for each realization of the system is not fixed 
by the value of a. Consequently the average value of the 
variables (s) and the second moment (s^) for a single 
realization of the system are described as s = TO((s))iy + 
(1 — m)so and = Tn{{s'^))w + (1 — to)sq where {{. . .))w 
indicates the average of the constraint variables on the 
effective potential W{s,y) and sq indicates the average 
value of the free variables in the considered realization. 

We note here that the observable volume of solutions 
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V differs from V and is defined as 

V{P)^ / TT^s^TT 1= det n{mN X mN) 

wliere 7?. is a square random matrix of mN elements and 
m is a the fraction of constraint variables to be deter- 
mined by saddle point equations. The direct calculation 
of (log(T^)) can be approximated very well with 

(log(y)) w (log(y)) + log[det7^(miV x mN)\ (10) 

with TO given by the solutions of as we checked di- 
rectly. 

To test the calculations we compute numerically the 
volume of solutions of randomly generated linear equa- 
tions and compared it with our theoretical predictions. 
As already mentioned in the introduction, finding the vol- 
ume of a polytope is a #P-coniplete problem, therefore 
the direct calculation of the volume of solutions is re- 
stricted, in practice, to problems with a small number of 
equations and variables. To simplify the computations, 
the elements of the matrix A were taken as ±1 with prob- 
ability \ and = 0. The variables were constrained to 
be in (-L/2,L/2) 

The simulations were done using vinci [loj | and the re- 
lated Irs code which shows reasonable performance 
in time for dense systems of no more than M = 4 equa- 
tions and A'^ = 12 variables. Moreover, one must keep 
in mind that the volume computed by Irs in these not 
full dimensional polytopes is the projection on the lexi- 
cographically smallest coordinate subspacefT^. For the 
matrices we used, this is equivalent to rescale the volume 
by a factor where N is the number of variables. 

The comparison between the numerical results and our 
analytical predictions appear in figure ^ where we show 
the direct average over 100 instances of systems of equa- 



tions with N = 6 and 9 variables and a = 1/3 as a func- 
tion of L. The analytical results were obtained trough the 
substitution in equations HlOfl and of the solutions of 
the system of equations Q with S = S = 0. As can be 
immediately seen, the agreement between our simulation 
and the theoretical line is very good. This, despite the 
fact that our predictions are valid in the thermodynamic 
limit and the simulations may be performed only for very 
small systems. 

In conclusion, we have described the emergence of flexi- 
bility in a dense linear systems of equations of N real vari- 
able and M equations. This flexibility results from the 
appearance, for a < ac, of an exponentially large num- 
ber of states that satisfy the constraints imposed to the 
system. Our results prove also that for a = M/N > ac, 
the standard RS scenario holds and the system is not 
satisfiable, while for a < ac an exponential number of 
solutions are present and the symmetry of permutation 
of the variables is broken. In this case, a fraction m 
of the variables must be considered free while the rest 
are fixed by the constraints. Our analytical predictions 
were tested computing numerically the volume of random 
polytopes derived from dense system of linear equations 
showing that the derivation presented provides the value 
of the caracteristic volume of a random polytope. The 
problem addressed in this paper is stylized and general, 
extensions of this work would include both the study of 
the same model on diluted graphs (with different degree 
distributions) in order to better describe the metabolic 
networks and a detailed analysis of the Minority Game. 
In both cases we expect that a scenario similar to the 
one discussed above holds. Work in these directions is in 
progress. 
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